feat(data): schema registry + default-off drift guard for tushare cache - #57
Merged
Merged
Conversation
Add a declarative endpoint schema registry and a default-off, report-only (or strict) drift guard that checks the RAW INPUT boundary of the tushare read-through cache, BEFORE parsing. Each `_parse_*` returns a fixed canonical column set and NaN-fills any canonical column whose source column is absent, so the stored `fields_hash` is immune to tushare changing its source columns. The real silent-corruption risk is a missing/renamed SOURCE column -> parser fills NaN -> cache stores NaN with zero signal. The highest-value check is therefore on the raw frame. - data/cache/schema_registry.py: EndpointSchema + REGISTRY (one per endpoint, canonical columns/keys derived from tushare_specs; expected_canonical_hash uses the ledger's _fields_hash), SchemaDriftFinding, and a stateful SchemaGuard with 4 checks: #1 missing required source cols (HARD), #2 unknown extra source cols (WARNING, only outside required/optional/known_extra), #3 parsed canonical mismatch (HARD), #4 stored schema hash changed vs ledger history (HARD, deduped once/endpoint/run; a migration detector, prior=None is always clean). strict mode raises a secret-free RuntimeError on HARD. Per-endpoint required/optional/known_extra follows one documented rule: REQUIRED = tushare guarantees it (axis/raise, fields= request, or standard core); OPTIONAL = defensively filled and legitimately omittable (e.g. index_member_all level names, guarded in the direct feed path); known_extra = documented-but-ignored response columns so check #2 stays quiet on them. Registry/endpoint sync uses an explicit RuntimeError (not assert, -O safe). - data/cache/tushare_cache.py: keyword-only schema_guard=None ctor arg; a single _guarded_parse wrapper at all 4 parse sites (None-guard => exact passthrough); schema_findings()/schema_summary() accessors. - data/cache/coverage.py: read-only additive last_fields_hash(endpoint). - qt/config.py: default-off SchemaGuardCfg on data.cache.schema_guard. - qt/pipeline.py: build the guard in _build_cache only when enabled; one secret-free summary line in _log_run_cache_stats. - tests/test_schema_registry.py: 62 network-free tests, incl. the forward lock (required cols sufficient, no NaN), the inverse lock (every optional col is droppable without raising), known_extra vs uncatalogued-column #2 behaviour, and the default-off byte-identical passthrough. The guard never sees a token or a data value (only column + endpoint names). Default-off => every existing config validates unchanged and the cache parse sites are byte-identical; phase0 anchor stays ic_mean=0.9600/annual=0.8408.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Data-layer: schema registry + default-off drift guard for the tushare cache
Next D-series step. Turns the cache's already-stored-but-unvalidated
fields_hashinto an explicit, default-off, opt-in schema drift guard.Core insight (why the guard sits at the raw-input boundary)
Every
_parse_*indata/cache/tushare_parsers.pyhardcodes a fixed canonical column set and defensivelyfillnas missing columns. So the storedfields_hashis immune to tushare changing its source columns — the real corruption risk is: tushare removes/renames a source column a parser depends on → the parser silently fills NaN → the cache stores NaN with zero signal. The guard checks the raw frame before parsing.What it does (all opt-in;
data.cache.schema_guard.enabled=falseby default)data/cache/schema_registry.py: per-endpointEndpointSchema(required / optional / known_extra source columns, canonical columns, natural key, expected canonical hash) derived fromtushare_specs.py(single source of truth), plus a statefulSchemaGuard.known_extra_columnscatalogue), feat(runtime): P2-2 execution realism — direction-aware fills + min_listing_days #3 parsed canonical ≠ registry (HARD), docs: CLAUDE.md progress — PR #1/#2/#3 merged #4 storedfields_hashchanged vs ledger history (HARD, deduped once/endpoint/run — a migration detector).report_only(default when enabled — collect findings + one secret-free summary log line) andstrict(opt-in — HARD finding raises before upsert, so the gap is NOT recorded as covered = retryable, mirroring the existing "failed fetch not recorded" semantic).qt/pipeline.py::_build_cache(guard built only when enabled; default-off ⇒None).data/cache/coverage.pygains a read-onlylast_fields_hash(endpoint)accessor.Invariants
_guarded_parse(None)is an exactparse(raw)passthrough; every pre-existing cache test passes unchanged; factor/alpha/portfolio/runtime/qfq/coverage semantics untouched; phase0 anchoric_mean=0.9600, annual_return=0.8408..config.jsonpath, no data values.TushareCacheonly; intraday schema guard deferred.Acceptance (independently re-run)
pytest: 758 passed (+62 schema tests; full-suite delta +62 over the 696 baseline)ruff: clean · all 19config/*.yamlvalidate ·run-phase0:0.9600 / 0.8408python -O: registry↔ALL_ENDPOINTS sync guard holds (assert→raise, survives optimization).config.jsonreferencesReview (2 rounds, replacing the prior external reviewer)
index_member_alldeclared SW level names as required → would loop onRuntimeErrorin strict for stocks where tushare omits a level column (proven by the direct path's own guard). Now required ={in_date}only; level names → optional (WARNING).fields=endpoints. Added aknown_extra_columnscatalogue so it warns only on a genuinely-unknown column.assert→raise(-O-safe); inverse droppability lock test; check feat(runtime): P2-2 execution realism — direction-aware fills + min_listing_days #3/docs: CLAUDE.md progress — PR #1/#2/#3 merged #4 secret-absence test assertions.Known follow-up (not in scope here)
Guard is wired into the pipeline/backtest cache path, not the live
data-updatewarm path (qt/data_updater.py) — that is the natural place to enable drift detection next.